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[57] ABSTRACT 

A graphical user interface for a computer system that 
includes one or more intelligent assistants. The interface 
includes composite icons comprising a graphical represen- 
tation of a human figure, a representation of a desk, and a 
mini-icon that associates an assistant with the object it 
supports. The assistant's desk can be opened to show its 
contents and the human figure can move from its position 
next to the desk to a position on the computer display screen 
next to a suggestions dialog box that displays suggested 
actions to the user. 

7 Claims, 7 Drawing Sheets 
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GRAPHICAL END-USER INTERFACE FOR 
INTELLIGENT ASSISTANTS 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to computer sys- 
tem end user interfaces and more particularly to a graphical 
end user interface that includes intelligent agents or assis- 
tants. 

2. Description of the Prior Art 

In certain currently existing computer systems with 
graphical user interfaces, an environment is created that is an 
electronic metaphor for a workplace or desktop. Objects 
found in the electronic workplace are presented for users to 
manipulate in ways that are similar to the way that they are 
manipulated in the real workplace. Objects, such as file 
cabinets, folders, documents, and printers, are displayed on 
the display screen as icons. Users manipulate these icons 
with the mouse to perform desired operations. For example, 
to file a document in a folder that is located in a file cabinet 
in the real workplace, the user opens the file cabinet, locates 
and opens the correct folder, and puts the document inside 
it In the electronic workplace, the user performs a similar 
process. The user opens the file cabinet, locates the correct 
folder, and drops the document icon on the folder icon. 
Users have been able to use their knowledge of a real 
workplace to perform this operation. 

While the workplace or desktop metaphor provides cer- 
tain advantages to the user in terms of intuitiveness and ease 
of use, many have suggested that the next step in personal 
computing will move beyond the desktop metaphor and its 
arrangement of passive objects toward a theater metaphor of 
active agents. In the theater metaphor, work is delegated to 
animated intelligent objects that operate like secretaries or 
assistants. 

Commercialization of these ideas has just begun. For 
example, Hewlett Packard Company has introduced 
"agents" in its New Wave product. However, these agents 
are not truly intelligent Each agent is merely a macro- 
capturing facility, which records and plays back a sequence 
of commands as performed by the user. This ability to replay 
what is called a "script" is the lowest form of artificial 
intelligence. It includes no ability for deductive inference or 
inductive discovery. In the New Wave product, the user 
records an action sequence by simply performing the action, 
but any subsequent variablization of the macro must be 
programmed by the user, if the user knows the macro 
language. The agent of the New Wave product is still 
passive; the script is activated only when the user gives it 
another object to work on, such as a spreadsheet to be 
formatted and printed The next logical step in artificial 
intelligence is to add a rule-based inferencing system. There 
has been some activity to build such intelligent office 
systems and interfaces and it appears that such work has 
been and will soon be commercialized as an electronic mail 
function. 

There is no question that rule-based agents are required 
for intelligent office systems. By definition, rules are 
required for corporate policies and regulations. The enter- 
prise needs to define that certain objects cannot be printed, 
for example. Rules are also the only way for the user to give 
instructions before going on vacation; there is no behavior to 
learn-by-example because the user is not around Simple 
rules can also be appropriate and well-formed by the user to 
handle many trivial jobs, such as automatic sorting of mail 
under appropriate key words. 
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On the other hand, the difficulties of rule expression, 
rule-based consistency, and a host of well known knowledge 
engineering problems suggest the need for case-based rea- 
soning and an adaptive system that can automatically learn 

5 by example. The user's own behavior provides the defacto 
validity and completeness and does not require the end 
user's explicit activities in knowledge engineering. Auto- 
mated knowledge acquisition is needed for all expert 
systems, but the many problems of knowledge acquisition 

10 are exacerbated if the general office worker is asked also to 
become a knowledge engineer. 

In co-pending U.S. patent application Ser. No. 07/720, 
278, filed Jun. 24, 1991, entitled CONVOLUTIONAL 
EXPERT NEURAL SYSTEM, now U.S. Pat No. 5357, 

15 597, assigned to the assignee of the present application, the 
disclosure of which application is incorporated herein, there 
is disclosed an adaptive neural system that exhibits continu- 
ous and instant learning, and a perfect memory. The system 
of application Ser. No. 07/720,278, has the ability to observe 

20 inputs and outputs, or conditions and results, and derive the 
rules that determine the outcome or result of various sets or 
combinations of inputs or conditions. Thus, the system of 
application Sen No. 07/720,278, promises to be an effective 
engine for intelligent assistants or agents. 

25 The suggestion that agents or assistants will appear as 
"little people" on the screen has already been made; 
however, little has been actually done to establish the display 
and metaphor of its explicit interface architecture. The 
graphical portrayals in other systems agents are still primi- 

30 rive. For example, the New Wave product uses the simple 
bust of a "secret agent", without any animation or multiple 
operations. Moreover, no system has adequately dealt with 
the problem of presenting different forms of intelligence 
within a graphical metaphor for the user. Various forms of 

35 intelligence are possible and should be used together, but 
this raises problems about how such different forms can be 
represented. 

SUMMARY OF THE INVENTION 

40 

The present invention provides a method of and system 
for displaying a set of assistants and their respective objects 
of responsibility together with their respective function and 
control and a method and system for displaying and invok- 

45 ing inferenced suggestions. All assistants are collected and 
managed within a "Bullpen." The workplace or desktop 
contains an "Assistants" object or icon, which can be opened 
to show the set of assistants. Each assistant is associated 
with an object Each assistant icon includes a graphical 

50 human figure and a graphical representation of a desk that 
has thereon an iconic representation of the object that the 
assistant serves. For example, in the preferred embodiment 
one assistant handles documents, as shown by a document 
object icon placed on the assistant's desk. The metaphor 

55 used is that each assistant is a software robot, each with its 
own desk, and with its responsibility demonstrated by the 
object on its desk. Global properties are also included in the 
bullpen. Global properties are those that are not given to 
only one object but are displayed to the separate objects in 

60 the bullpen. For example, in the preferred embodiment a list 
of facts, such as the user's name, position code, vacation 
time, and the like, is displayed in a bulletin board object 
Each assistant keeps its own set of rules, but all the assistants 
use the same set of global facts which the user can maintain 

55 in one place. 

In the present invention, the assistant's "learning" is 
located within its head. The adaptive form of intelligence is 
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provided by neural network methods, and as such is located BRIEF DESCRIPTION OF THE DRAWINGS 

within the assistant's "hrain." Clicking on the head of the ^ ^ f ^ ^ ^ k 

assistant's graphical human figure opens it to the neural rMXJ - A " J~ \ ™» ■ j ,/ ^Jl^L 

representationof what the assistant has learned about objects screen with a WORKPLACE window display thereon 

of the particular class. The user can view the relevant 5 including an Assistants icon. 

variables for the assistants actions and exercise "what-ifs." FIG. 2 is a pictorial view similar to FIG. 1 but including 

User developed rules for each object are located in the an "Assistants Bullpen" window, 

assistant's desk. Clicking on the desk opens it to a window piQ, 3 is a pictorial view similar to FIG. 2 in which an 

that contains its contents. Rules are not learned, but are assistant icon is enlarged. 

specifically stated to each assistant To rnaintain separation 10 pictorial view similar to FIG. 2 but including 

of this knowledge from that which is learned, such rules are MAaaataaA Desk' window 

characterized as "instructions* to the assistant from the an Assistant uesjc winaow. 

system user and are kept as a document or list object within FIG. 5 is a view similar to FIG. 2 but including an 

the assistant's desk Enterprise regulations are also located "Inbasket" window and a "Document" window, 

in the desk Regulations, like user instructions, are not FIG. 6 is a pictorial view similar to FIG. 5 including a 

learned but are by definition simply stated as rules. This "Suggestions" window. 

affords an object-oriented place for a corporate enterprise to FIG 7 is a pictorial view of a a neural network window 

establish rules and an easy to use metaphor for the user to a computer display screen. 

find and reference them. Performance reports and manage. w ^ fl , L1 7,, F ' . . . . 

ment controls may also be located in the desk This is a FIG. 8 is a block diagram representation of a computer 

metaphor for an employee's performance record as a docu- 20 system for practicing the method and system of the present 

ment kept in a real desk. The record of performance for both invention. 

the neural network and the user-defined rules, such as FIGS. 9-12 comprise a flow chart of a preferred software 

frequency of use and inferencial decisions versus actual implementation of the method and system of the present 

outcome, can be accessed within this object Threshold invention, 

setting, performance trade-offs, and the like can be managed 25 

from this information. DESCRIPTION OF THE PREFERRED 

Object-relevant vocabulary may also be kept within the EMBODIMENT 

desk As voice recognition is incorporated into the manipu- Referring now to ^ drawings, and first to FIG. 1, a 

lation of object-onented interfaces, a proper metaphor for its * f designated generally by 

use should be included. !?« ^ 3 ° * 1L ^ screen 11 has^played therein a 

broadcast a command such as CaU John Adams and the Window 13 includes a title bar 15, an action bar 

appropriate assistant (in this case the phone assistant) will ^ ^ M bars 21 and ^ ^ of which 

then take the commanded action rather than the user saying > ^ ^ a Mw bQR|fir ^ 

something like 'Telephone, call John Adams. In natural . t 

interaction, people do not speak to inanimate objects. 35 ™e bar 15 mchides in addmon .to a label that identifies 

Vocabularies are defined within the scope of a particular Wow 13 as me 'Workp^ 

object. The definition of that vocabulary, which are those 27 and sizing icons 29 and 31. System menu icon enables a 

commands that the assistant can understand and respond to, user to display a pull-down menu of actions that can be 

can be made here performed on the window itself rather than the object or 

In the present invention, each assistant can make recom- 40 objects displayed in the window. For 

mendations or suggestions that can be requested from the « system menu icon 27 to move or close window ML 

action bar of the serviced object window. For example, Sizing icons 29 and 31 provide a convenient way fortheuser 

JSSSn L context of a particular mail document, thesis- "> OT »»— respectively, window -13. 

tant can be called, whereupon it will leave its desk in the Action bar 17 includes a list of actions that can be 

bullpen and then appear floating next to the opened docu- 45 performed on the object or objects in window 13. The user 

ment holding a list of actions. The assistant is not a normal can move a pointer 33 with a mouse (not shown) to select 

object within a presentation window. Rather, it is separate as various objects in window 13 and then invoke various 

a special type of object that can traverse the system or actions from pull-down menus that can be ^displayed beneath 

screen. In me preferred embodiment, each assistant can learn the items of action bar 17. Client area 19 of window 13 is 

whether or not to display its suggestions automatically. 50 populated with a plurality of objects represented by icons. In 

When the user asks for suggestions from the action bar, the the workplace environment, the objects are electronic meta- 

user is committing an action, like any other, that the assistant phors for objects that a user would find in a real workplace, 

can learn to predict If the assistant learns the conditions For example, the workplace environment includes a calen- 

under which the user will request its suggestions, the assis- dar icon 35, which represents an electronic calendar object 

tant can automatically present itself when a proper instance 55 or application. The user can open calendar icon 35 and use 

of the object is opened. Clicking on the assistant's feet will it in ways similar to the way people use real calendars, 

cancel its presence. The assistant's feet are, in effect, its Similarly, the workplace environment includes aprintencon 

"cancel" button. Clicking on its feet will cause the sugges- 37. The user can, by direct manipulation, use the mouse and 

tion list to disappear and the assistant to return to its desk pointer 33 to move a graphical representation of a document 

Clicking on any item of the list of suggestions will cause that 60 to printer icon 37 and thereby cause the document to be 

one suggestion to be selected and highlighted. In the pre- printed. 

f erred ernbodiment, the assistant icon has displayed adjacent Workplace window 13 includes, in addition to normal 

to its head a quotation bubble with the word "Okay?" in it office equipment objects, such as calendar 35 or printer 37, 

Clicking on the quote bubble will invoke all selected actions an assistants object represented by assistants icon 39. As will 

in the list of suggestions and the assistant will return to its 65 be explained in detail hereinafter, the system of the p-esent 

desk. Clicking on the assistant's head will display its current invention includes a plurality of intelligent assistants that are 

reasoning. assigned to various objects in the electronic workplace. The 
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assistants have the capacity to learn rules regarding how the user-defined rules, such as frequency of use and infer- 

various objects are to be handled, either by observation of ential decisions versus actual outcome. The performance 

how the user handles the objects or by receiving instructions data provides a means by which the performance of the 

or regulations from the user or the enterprise for which the system may be evaluated and enhanced. Vocabulary icon 63 

user works. Preferably, the assistants include neural net- 5 represents object-relevant terms to which the assistant 

works of the type disclosed in co-pending U.S. patent responds. As voice recognition is incorporated into the 

application Ser. No. 07/720,278, filed Jun. 24, 1991, entitled manipulation of object-oriented interfaces, each assistant 

CONVOLUTIONAL EXPERT NEURAL SYSTEM, will have a set of vocabulary that it can understand and 

assigned to the assignee of the present application, the respond to. 

disclosure of which application is incorporated herein. Q Referring now to FIG. 5, the user has opened an Inbasket 

Referring now to FIG. 2, there is shown in addition to window 65, In-basket window 65 was preferably opened by 

workplace window 13, an Assistants Bullpen window 41. double-clicking on Inbasket icon 67 of Workplace window 

Assistants Bullpen window 41 is similar to window 13 in 13. la„basket window 65 contains incoming electronic mail 

that it includes a title bar, an action bar, and scroll bars. documents that the user has received. In FIG. 5, the user has 

However, the client area 43 is populated with a plurality of selected to view document 69 in In Basket window 65 and 

assistants icons., including, for example, a document assis- has opened a Document window 71. 

tant icon, which is designated generally by the numeral 45. DoC ument window 71 includes an action bar 73 that 

Document assistant icon 45 includes a graphical represen- "Assistant" action item 75. In the present 

tation of a human FIG. 47 and a mini-desk 49 which has ; ; . . . t . . . 

a- 1 a * 7 . . 0 V^i^nt invention, whenever an object has an assistant assigned to it 

fashion, each of the other assistant icons in Assistants object's window. In FIG. 5 the user has selected Assistant 
Bullpen window 41 include a graphical representation of a item 75 and the system has displayed an assistant pull-down 
human figure in a mini-desk with an iconic representation of menu 77. Assistant pull-down menu 77 includes a Sugges- 
the object that the assistant supports. Additionally, Assistants tions item 79, as well as the items that are contained in the 
Bullpen window 41 includes a bulletins object that is 25 assistant's desk. Documents assistant 45 has "reviewed" the 
represented by a Bulletins icon 51. Bulletins icon 51 can be document of window 71 and it has, either by having 
opened to display a window into which the user can input observed the user's past actions or by reference to instruc- 
facts that the assistants may need in order to perform their tions or regulations, formulated a set of actions that the user 
functions. For example, if the user is going to be out of the is likely to perform on the document. The assistant may 
office on vacation, the user can identify his or her backup 3Q automatically cause the system to perform certain actions 
and tell all assistants what dates the user will be out of the " on its own" without input from the user; however, in the 
office. As those skilled in the art recognize, windows can be preferred embodiment of the invention, the assistant non- 
opened by double clicking on an icon, or selecting an icon mally does not cause the system to perform those actions 
and then selecting the open action from the pull-down menu without the concurrence of the user. Thus, those actions are 
associated with the "File" item of action bar 17. 33 presented to the user as suggestions. If the user wishes to 
Referring now to FIG. 3, human FIG. 47 of document look at the suggested actions, the user may select sugges- 
assistant icon 45 is shown enlarged. The figures of the tions item 79 with pointer 33 and thereby cause the system 
assistant icons can be enlarged by clicking pointer 33 on the to display a Suggestions dialog box 81, as shown in FIG. 6. 
feet 53 of the figures. The human figures may be enlarged to Referring to FIG. 6, Suggestions dialog box 81 is shown 
provide more functionality to the interface. For example, a ^ as if human FIG. 47 is holding it In FIG. 6, human FIG. 47 
user might obtain help by clicking on the figure's shoulder has left its position next to mini desk 49 in window 41. FIG. 
and dismiss the help by clicking on the figure's mouth. 47 has displayed next to its head a speech bubble 83 that has 
Whenever the assistant icon is in the Assistants Bullpen the word "Okay?" in it Speech bubble 83 functions in the 
window 41, clicking on the figures feet will toggle the figure same way as the "Okay?" pushbutton in a normal dialog 
between its large size and its small size. The metaphor is that 45 box. However, speech bubble 83 provides a more anthro- 
of the assistant coming to attention and going back to his pomorphic feel to the interface. 

desk. Suggestions dialog box 81 contains a list of suggested 

Referring now to FIG. 4, there is shown, in addition to actions, including "callSender" and "rKintLocal.'' Thus, the 

workplace window 13 and Assistants Bullpen window 41, document assistant has suggested that the user call the 

an Assistant Desk window 55. Assistant Desk window 55 is 50 sender of the document and print the document locally, 

preferably opened by double clicking pointer 33 on mini- Again, the assistant waits for the user's concurrence before 

desk 49 of document assistant 45. Assistant Desk window 55 performing the action. However, in the preferred 

includes various objects, including instructions 57, regula- embodiment, the assistant may take certain actions to facili- 

tions 59, performance 61, and vocabulary 63. Instructions tate performance of the action. Far example, since one of the 

icon 57 represents user defined rules provided to the assis- 55 suggestions is callSender, the assistant may look up 

tant. Instructions are not learned through observation, but "Frank's" (the sender) telephone number and perform other 

are simply given to the assistant For example, the user may prerequisite activities to enhance system performance. The 

instruct the assistant always to print correspondence from user can cause the system to perform the suggested actions 

the user's immediate supervisor. by selecting the various suggestions with pointer 33. If the 

Regulations icon 59 represents enterprise regulations. For 60 user desires to have the system perform the selected sug- 

example, it may be an enterprise regulation that documents gested actions, the user can do so by clicking on speech 

carrying a certain security classification cannot be printed on bubble 83. If the user does not want to have the system 

printers located in uncontrolled spaces. Such regulations are perform any of the suggested actions, the user may cancel 

made available to the assistant, perhaps as a constraint the suggestions by clicking on the feet 85 of human FIG. 47. 

against any other inference. 65 Thus, feet 85 are equivalent to the "cancel" pushbutton of a 

Performance icon 61 represents an object that contains a normal dialog box. Suggestions dialog box 81 will remain 

record of the performance for both the neural network and on screen 11 until the user either cancels it by clicking on 
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feet 85 or performs the selected suggested actions by click-. system tests at decision block 145 whether or not perfor- 
ins on speech button 83. When suggestions dialog box 81 is mance is selected. If it is. the system opens a performance 
dismissed, human figure 47 returns to its position next to window at block 147 and returns. If P^rnwnce is not 
mini-desk 49 of assistants bullpen window 41. selected, men the system tests at «^ 149«tote 

, t . c or not vocabulary is selected. If it is, then the system opens 

ff the user desires to understen^ the vocabiilary Wow at block m and returns. FinaUy, if 

the suggestions of dialog box 81. the user may dick on the vocabul ^ not selected then the system returns as indi- 

head 87 of FIG. 47, which will cause the system to display t > continue t0 monitor user mput at block 111 

a neural network window 89. as shown in FIG. 7. Referring at fjq 9 

to FIG. 7, neural network 89 shows two neurons, which — uger a 

indicate how the assistant amved at its suggests to call 10 ^^^fjf^^^ toe sys^xtx tests at deeision blodc 153 

the sender and print the document locally. ^ or not an other object is selected. If it is. the system 

Referring now to FIG. 8, there is shown a block diagram g ^ for me otnex object at block 155. Then, 

of a system according to the present invention. The system &e gystem tests at block 157 whether or not the 

includes a processor 91, which includes a central processing selected object is assisted. If it is not, the system returns to 

unit (CPU) 93 and random access memory 95. The system block n to continue mom toring user input If the object is 

may also include additional memory in the form of a hard assisted, then, at block 159, the system includes an assistant 

disk storage device 97. a floppy disk device 99, or, item m ^ acdofl bar for me ob ject window. Then, the 

preferably, both. Floppy disk device 99 is adapted to receive m tMte ^ ^ciston b i oc k 161 whether or not the user 

a diskette 101 that may have recorded thereon the software ^ ^ for suggesdons m suc h cases. If the assistant has 

implementation of the present invention. The system of FIG. leaned that the user does ask for suggestions in such cases, 

8 includes user interface hardware, including a display 103, me system automatically opens the suggestions dialog box at 

a mouse 105, and preferably a keyboard 107. The system Wock 163 ff ^ usef does not norma uy ask for suggestions 

may, optionally, include a printer 109. m guch cases ^ ^ S y Stem tests at decision block 165 

Referring now to FIGS. 9-12. there is shown a flowchart ^ wne ther or not the assistant item is selected from the action 

of a preferred software implementation of the present inven- bar. If it is not, the system returns to block 111 to continue 

tion. Referring first to FIG. 9, the system first begins the monitoring user input If the assistant item is selected, then 

desktop or workplace as indicated generally at block 109. me system displays the assistant pull-down menu at block 

After the desktop is running, the system monitors user input ^7 ^ wa its for user input. If, at decision block 169. the 

at block 111. If the system receives a user input, the system ^ user s elects suggestions, then the system opens the sugges- 

tests, at decision block 113 whether or not the assistants tions dialog box at block 163. If the user does not select 

object has been selected. If it has, the system opens the suggestions, then the system goes through the routine indi- 

assistants bullpen window at block 115 and then proceeds to cate 4 generally by "B", which is shown in and described in 

the portion of the flowchart indicated generally by the letter connection with FIG. 11. 

"A", which is shown in FIG. 10. 35 Referring still to FIG. 9, after the system has opened the 

Referring to FIG. 10, the system tests at decision block suggestions dialog box at block 163, the system proceeds as 

117 whether or not the bulletins object is selected. If it is, the indicated generally by the letter "C" , which is shown in FIG. 

system opens the bulletins window at block 119 and returns. 12. Referring to FIG. 12, the system displays any sugges- 

If the bulletins object is not selected at decision block 117 tions in the suggestions dialog box at block 165. Then, the 

then the system tests at decision block 121 whether or not an 40 system tests at decision block 167 whether or not the 

assistants feet are selected. If the feet are selected, then the assistant bullpen window is open. If it is, the assistant icon 

system tests at decision block 123 whether or not the is erased from the bullpen window at block 169. After the 

assistant is enlarged. If the assistant is enlarged, then the assistant icon has been erased from the bullpen window, or 

human figure representation of the assistant is reduced at if the bullpen window is not open, the system displays the 

decision block 125. If the assistant is not enlarged, then the 45 assistant icon next to the dialog box at block 171 and 

system enlarges the assistant figure at block 127 and returns. displays the "Okay?" speech bubble next to the assistant 

Hius, in FIG, 10, the user can toggle between a reduced icon at block 173 and monitors user input If, at decision 

assistant and an enlarged assistant by selecting the assis- block 175. the user input is the selection of a suggestion, the 

tant's feet If at decision block 121, the feet are not selected, selected suggestion is highlighted at block 177 and the 

then the system tests at decision block 129 whether or not 50 system returns to continue monitoring user input If, at 

the assistant image's head is selected. If it is, the system decision block 179, the user selects the head of the assistant 

opens a neural network window at block 131 and returns. If icon, the system opens the neural network window at block 

the head is not selected at decision block 129, then the 181 and returns to continue monitoring user input. If. at 

system tests at decision block 133 whether or not the decision block 183, the user selects the "Okay?" speech 

mini-desk is selected. If it is not then the system as indicated 55 bubble, the system performs all selected suggestions at 

at "D" returns to continue monitoring user input at block 111 block 185. If the user input at decision block 183 is not 

of FIG. 9. If, at decision block 133, the mini-desk is selected, selection of the "Okay?* speech bubble, then the system 

the system opens the assistant desk window at block 135 and tests at decision block 187 whether or not the assistant's feet 

proceeds to the portion of the flowchart indicated by the are selected. If they are not the system returns to continue 

letter "B", which is shown in FIG. 11. 60 monitoring user input If me feet are selected, at decision 

Referring to FIG. 11, the system tests at decision block block 187 or if okay has been selected, as described above, 

137 whether or not instructions are selected and if so, the the system closes the dialog box and erases the assistant icon 

system opens the instructions window at block 139 and and "Okay?" speech bubble at block 189. Then, the system 

returns. If instructions are not selected, the system tests at tests at decision block 191 whether or not the assistant 

decision block 141 whether or not regulations are selected. 65 bullpen window is open. If it is, the system displays the 

If they are. then the system opens a regulations window at assistant figure icon in the bullpen window at block 193 and 

block 143 and returns. If regulations are not selected, the returns to block 111 of FIG. 9 to continue monitoring user 
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input. Finally, refeiring to FIG. 9, if the user input is mined utilizing said neural representation of learned expe- 

something other than selection of the assistants icon or rience with respect to actions within said computer system 

another object the system processes such other user input at related to said at least one object 

block 195 and continues to monitor user input at block 111. 4. The method according to claim 2, further including the 

From the foregoing, it may be seen that the present 5 step of displaying a graphic representation of a human figure 

invention provides a useful end user interface for a system having a head adjacent to said pull-down menu containing a 

of intelligent assistants or agents. While the invention has suggested action and a speech bubble adjacent to said head 

been particularly shown and described with reference to a and wherein said step of performing said suggested action in 

preferred embodiment, those skilled in the art will under- response to a selection thereof comprises the step of per- 

stand that various changes may be made in form and detail w forming said suggested action in response to selection of 

without departing from the spirit and scope of the invention. said speech bubble. 

What is claimed is: 5* T" e me *h°d according to claim 4, further including: 

1. A method for performing a selected action within a selecting the head of said human figure; 

computer system having a display screen and a graphic automatically opening a neural network window in 
pointing device wherein said computer system is capable of 15 response to selecting said head; and 
performing a plurality of diverse actions, said method com- displaying in said neural network window a graphic 
prising the computer implemented steps of: representation of a neural network associated with said 
associating an intelligent assistant with at least one object intelligent assistant 
within said computer system, said intelligent assistant $ # A method for performing a selected action within a 
including a neural representation of learned experience 20 computer system having a display screen and a graphic 
with respect to actions within said computer system pointing device wherein said computer system is capable of 
relating to said at least one object; performing a plurality of diverse actions, said method corn- 
displaying an object window within said display screen of prising the computer implemented steps of: 

said computer system in response to selection of said at 25 displaying at least one assistant icon within said display 

least one object by a user utilizing said graphic pointing screen, said at least one assistant icon comprising a 

device, said object window including an action bar; mini-icon representative of a selected object, a graphic 

displaying an assistant item within said action bar within representation of a human figure symbolizing an intel- 

said object window in response to said association of Hgent assistant having a neural representation of 

said intelligent assistant with said at least one object; 30 learned experience with respect to actions within said 

and computer system related to said selected object, and a 

selectively displaying a pull-down menu containing a representation of a desk displayed adjacent to 

suggested action among said plurality of diverse said human figure; 

actions in association with said assistant item, said opening an object window for said selected object asso- 

suggested action determined utilizing said neural rep- 35 ciated with said intelligent assistant in response to a 

resentation of learned experience with respect to selection of said mini-icon; and 

actions within said computer system related to said at displaying a suggestion dialog box in association with 

least one object. said object window, said suggestion dialog box con- 

2. The method according to claim 1, further including the taining a suggested action among said plurality of 
step of rjerforming said suggested action in response to a 40 diverse actions, said suggested action determined uti- 
selection thereof by said user within said pull-down menu. lizing said neural representation of learned experience 

3. The method according to claim 1. wherein said step of with respect to actions within said computer system 
selectively displaying a pull-down menu containing a sug- related to said selected object 

gested action among said plurality of diverse actions in 7. A method according to claim 6, further including the 

association with said assistant item comprises the step of 45 step of performing said suggested action in response to a 

selectively displaying multiple suggested actions among selection thereof by a user within said suggestion dialog box. 
said plurality of diverse actions in association with said 

assistant item, each of said multiple suggested actions deter- * * * * * 
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